1. Field of the Invention
The present invention relates to the estimation of gradients. More particularly, the present invention relates to the use of noise injection and correlation in analog circuits to estimate gradients.
2. Art Background
Computation of gradient descent and related descent and annealing techniques is frequently performed in digital systems and used in a variety of applications. In some applications, such as neural networks, it is desirable to utilize a gradient descent method of optimization to minimize an objective function.
One common approach to optimization is gradient descent wherein, for example, in one dimension: ##EQU1## where .alpha.&gt;0 and is a constant. Because the value of f decreases until x stops changing: ##EQU2## At the point df/dx=0, the point is at a minimum.
For example, the one dimensional gradient descent method of optimization may be used as part of an adaptive control system where a function f(x) is bounded below and f and x are both scalars, and it may be necessary to minimize the function where .function. represents an error between actual and desired outputs. Alternately, it may be necessary to minimize a function as part of a learning system (such as a neural network).
One application of this capability to estimate gradients and perform descent is to automatically set circuit parameters. Automated on-chip parameter setting has become an important component of analog VLSI implementations of learning. Off-chip optimization techniques, such as using a digital computer, become impractical in implementation as the number of parameters increases. This is due to the fact that when extended to multiple dimensions, searches in large dimensional spaces are difficult and slow to perform in digital computer implementations. However, while the gradient descent algorithm is straightforward to implement on a digital computer, exact computation of the objective function's gradient is very difficult to implement in analog circuits, including analog VLSI.